Omissions are a major risk in testing. Test planning techniques are a well known way to improve the quality of testing by reducing both omissions and redundancy. These techniques try to prevent omissions in advance (i.e., before tests are implemented and executed) by selecting what to test out of a possibly enormous test space, in a way that reduces as much as possible the risk of bugs escaping to the field. One such known technique is Combinatorial Test Design (CTD).
Functional coverage analysis techniques are another well known way to detect omissions and increase the quality of testing. Coverage analysis measures how well executed tests covered the test-space. Coverage analysis measures the execution of tests against a functional model of the System Under Test (SUT) and reports which aspects are missing and which are covered by the executed tests.
Both test planning and functional coverage analysis require modeling of the test space as a functional coverage model. The test space can represent inputs, scenarios, configurations, the application's internal state, or any other aspect that one is interested in testing. Traditionally, the functional coverage model consists of a set of functional attributes, respective domains of values for each attribute, and restrictions on the value combinations. The restrictions may be Boolean expressions defining when value combinations are valid or invalid. Restrictions may be represented in other manners as well, such as by explicitly enumerating all valid or all restricted combinations of values. The set of valid value combinations, i.e. all combinations from the Cartesian product of the values that pass the restrictions, define the space to be tested.
A functional coverage is measured with respect to a functional coverage model. The functional coverage model defines a triplet: functional attributes, a domain for each functional attribute, and a set of restrictions. The functional attributes may be any attribute of the SUT, such as for example a type of web browser being used, an underlying Operating System, a number of threads operating, whether the output was printed. The domains may define for each attribute a set of possible values. For example, the web browser may be Microsoft® Internet Explorer®, Google® Chrome®, or Mozilla Firefox™. Similarly, the operating system may be Microsoft® Windows®, or Linux™. The cross-product of the different combinations of the attributes defines a functional coverage test-space. The test-space comprises a set of coverage tasks, each representing functional requirements to be tested: one requirement per functional attribute, and the requirement is that the functional attribute will exhibit the behavior of the value of the coverage task. The coverage task may be seen as a tuple of one value per attribute. In a functional coverage model in which there are three functional attributes, one having three possible values, the second having two possible values, and the third having ten possible values, the cross-product test-space comprises sixty (60) coverage tasks.
The functional coverage model may further comprise a set of restrictions defining a series of values of different attributes that may not appear together. For example, consider a functional coverage defining two attributes: ACTION and USER. The ACTION attribute may be each of the following values: RETRIEVE, STORE, and MODIFY PERMISSION. The USER attribute may be each of the following values: ADMIN, USER, GUEST. In some cases, a guest user cannot modify permission. A restriction may be defined to indicate that the couple (GUEST, MODIFY PERMISSION) is not a valid couple. The fact that a trace does not comprise an entry covering a coverage task that includes the couple does not affect the functional coverage. In other words, all possible coverage tasks—which together form the maximal possible coverage with respect to a functional coverage—do not include any coverage task that comprises the restricted couple. It will be understood that restrictions may be defined with respect to a combination of values of different attributes, and not necessarily with respect to a complete tuple of values. Furthermore, the restriction may be any constraint on combination of values, which may be represented using a Boolean formula over the functional attributes and associated values.